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ABSTRACT  ,  i 

A  new  penalty  function  is  associated  with  an  inequality  constrained 
nonlinear  programming  problem  via  its  dual.  This  penalty  function  is 
globally  differentiable  if  the  functions  defining  the  original  problem  are 
twice  globally  differentiable.  In  addition,  the  penalty  parameter  remains 
finite.  This  approach  reduces  the  original  problem  to  a  simple  problem 
of  maximizing  a  globally  differentiable  function  on  the  product  space  of 
a  Euclidean  space  and  the  nonnegative  orthant  of  another  Euclidean  space. 
Many  efficient  algorithms  exist  for  solving  this  problem.  For  the  case 
of  quadratic  programming,  the  penalty  function  problem  can  be  solved 
effectively  by  successive  overrelaxation  (SOR)  methods  which  can  handle 
huge  problems  while  preserving  sparsity  features.  ,/ 


AMS  (MOS)  Subject  Classifications:  90C30,  90C20 
Key  Words:  Nonlinear  programming,  quadratic  programming,  penalty 
functions,  SOR  methods 
Work  Unit  Number  5  -  Operations  Research 
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SIGNIFICANCE  AND  EXPLANATION 


The  problem  of  minimizing  a  function  of  several  variables 
subject  to  inequality  constraints  is  reduced  to  the  problem  of 
maximizing  a  smooth  function  subject  to  nonnegativity  constraints. 
The  latter  problem  can  be  easily  solved  by  many  known  efficient 
methods.  Very  large  quadratic  problems  can  be  solved  by  using 
successive  over-relaxation  methods  which  will  preserve  any  sparsity 
the  original  problem  may  have. 
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A  DUAL  DIFFERENTIABLE  EXACT  PENALTY  FUNCTION 


S.  -P.  Han  and  0.  L.  Mangasarian 

a 

1.  Introduction 

It  Is  well  known  that  exterior  penalty  functions  [6,13]  In  mathe¬ 
matical  programming  suffer  from  one  of  two  difficulties.  Either  the 
Hessian  of  the  penalty  function  becomes  ill-conditioned  as  the  penalty 
parameter  approaches  Infinity  [6,20],  or  the  penalty  function  Is 
nondifferentlable  [13].  There  have  been,  however,  attempts  at  obtaining 
penalty  functions  which  are  both  differentiable  and  for  which  the 
penalty  parameter  remains  finite  [8,3,4,!].  We  present  here  a  different 
and  an  extremely  simple  penalty  function  which,  by  taking  advantage  of 
the  structure  of  the  dual  problan,  results  In  a  penalty  function  which 
Is  differentiable  and  for  which  the  penalty  parameter  remains  finite. 

The  key  Idea  behind  the  present  approach  Is  extremely  simple  and  Is  best 
Illustrated  by  the  following  equality-constrained  minimization  problem 

minimize  f(x)  subject  to  h(x)  »  0 
xeR” 

where  f  and  h  are  differentiable  functions  from  the  n-dimensional 
real  Euclidean  space  r”  Into  R  and  R*^  respectively.  The  classical 
exterior  penalty  problem  for  this  problem  is 

minimize  f(x)  +  §-||h(x)||^ 

XcR"  ^ 

where  a  Is  a  positive  penalty  parameter  and  11*11  denotes  the  2-norm. 
At  stationary  points  of  the  penalty  problem  we  have 

7f(x)  +  a7h(x)^h(x)  *  0 

Sponsored  by  the  United  States  Army  under  Contract  No.  DAAG29-8D-C-0041 . 
This  material  is  based  upon  work  supported  by  the  National  Science 
,  Foundation  under  Grants  No.  MCS-790166  and  ENG-7903881. 
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where  Vf(x)  is  the  nxl  gradient  of  f,  Vh(x)  is  the  kxn  Jacobian 
of  h  and  the  superscript  T  denotes  the  transpose.  In  order  for  this 
condition  to  approach  the  stationarity  conditions  for  the  minimization 
problem,  which  are 

7f(x)  +  Vh(x)^u  =  0.  h(x)  -  0 

where  u  is  an  kxl  vector  of  Lagrange  multipliers,  the  quantity 
oh(x)  must  approach  u.  Since  h(x)  =  0,  it  turns  out  that  in  general 
o  must  approach  «.  There  are  exceptions.  For  example  if  u  »  0 
then  a  need  not  approach  «.  This  is  an  exceptional  case  which 
does  not  hold  in  general  for  the  original  minimization  problem. 

However,  if  we  consider  the  Wolfe  dual  [22,15]  to  an  inequality 
constrained  minimization  problem,  then  the  optimal  Lagrange  multiplier 
associated  with  the  equality  constraint  of  the  dual  is  zero  provided  that 
the  Hessian  of  the  Lagrangian  is  nonsingular  at  the  optimum.  Hence  for 
the  exterior  penalty  problem  associated  with  Wolfe  dual  we  can  show 
(Theorems  1  to  4)  that  under  rather  natural  conditions  the  penalty 
parameter  remains  finite.  Hence  we  can  obtain  a  globally  differentiable 
penalty  function  with  a  finite  penalty  parameter.  Because  our  penalty 
problem  formulation  depends  in  an  essential  manner  on  the  dual  problem, 
our  results  are  local  results  in  the  absence  of  convexity,  and  become 
global  results  if  convexity  is  assumed.  Because  our  penalty  function  is 
smooth  and  its  parameter  is  finite  it  has  important  computational 
implications.  For  example,  fast  methods  of  smooth  optimization  could  be 
used  to  directly  optimize  the  differentiable  penalty  function  (Algorithm  1), 
or  the  function  may  be  used  as  in  [12]  in  enlarging  the  convergence  region 
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of  fast  but  locally  convergent  algorithms  [9,11].  In  addition,  for 
positive  definite  quadratic  programming  problems,  our  penalty  function 
can  be  used  to  derive  a  successive  overrelaxation  (SOR)  algorithm 
without  the  need  to  invert  the  underlying  positive  definite  matrix  of 
the  problem  (Algorithm  2).  SOR  algorithms  have  proved  to  be  successful 
in  solving  linear  programming  problems  [17]  and  have  the  potential  for 
solving  enormous  problems  that  cannot  be  tackled  by  pivotal  methods 
while  at  the  same  time  preserving  the  sparsity  of  the  problem. 

Besides  this  Introduction*  this  paper  contains  two  sections.  In 
Section  2  we  treat  the  general  nonlinear  programming  problem  while  in 
Section  2  we  specialize  to  the  quadratic  programming  case  to  obtain 
sharper  results.  Section  1  contains  theorems  relating  stationary  points, 
local  and  global  optima  of  the  nonlinear  inequality  constrained  problem 
to  those  of  the  penalty  problem.  We  also  give  a  simple  gradient  projec¬ 
tion  algorithm  for  optimizing  the  penalty  function.  In  Section  3  we 
have  similar  results  for  the  quadratic  programming  case.  We  also  present 
an  SOR  method  for  quadratic  programming  which  is  a  generalization  of  the 
SOR  method  used  with  successful  computational  results  on  linear 
programming  [17]. 

We  briefly  describe  our  notation.  All  vectors  in  r”  will  be 
column  vectors  unless  transposed  to  a  row  vector  by  the  superscript  T. 
r”  will  denote  the  nonnegative  orthant  {x|x€R”,  x>0}.  For  x  in  r”, 

x^,  i»l . .  will  denote  its  ith  component,  while  x^.  will  denote  a 

vector  in  r”  with  components  (x^)^  »  max  {x^,0>,  1*1... .,n  and  |lx|| 
will  denote  the  Euclidean  norm  (x^x)**.  For  an  mxn  real  matrix  A,  A^. 
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will  denote  the  1th  row,  A*j  the  jth  column,  and  If 

Jc{1 . n}  then  Aj  will  denote  the  submatrix  with  rows  A^,  Id,  Aj 

will  denote  the  submatrix  with  columns  A*J,  jfj,  and  Ajj  will  denote 
the  submatrix  with  elements  A^j,  lei  and  JeJ.  For  a  differentiable 
function  f:R”-^R,  Vf(x)  will  denote  the  nxl  gradient  vector,  while 
for  a  differentiable  function  gtR^-t-R"*,  Vg(x)  will  denote  the  mxn 
Jacobian  matrix.  For  a  twice  differentiable  function  L:R*’ x 
Vj^L(x,u)  will  denote  the  nxl  gradient  with  respect  to  x,  7yL(x,u) 
will  denote  mxl  gradient  with  respect  to  u,  V  L(x,u)  will  denote 
(n+m)  X  (n+m)  Hessian  with  respect  to  both  x  and  u  whose  submatrix 
components  are  denoted  as  follows 


v\(x,u) 


7jjj^L(x.u)  7^yL(x,u) 


7yJjL(X.U)  7yyL(X,U) 


For  a  nonlinear  programming  problem  such  as  (1)  below,  a  point 
(x,u) 6  r”^  satisfying  the  Karush-Kuhn-Tucker  conditions  (!')  Is  said 
to  be  a  KKT  point,  while  x  Is  said  to  be  a  stationary  point  of  (1). 


Whenever  a  point  (x,u)  is  a  KKT  point,  the  differentiability  of  f 
and  g  at  x  Is  Implicitly  assumed. 


•>. .  -mr 
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2.  The  General  Nonlinear  Programming  Problem 
We  consider  here  the  problem 


minimize  f(x)  subject  to  g(x)  <0  (1) 

xcR” 

where  f  Is  a  function  from  the  n-dimenslonal  real  Euclidean  space  r” 
Into  the  reals  and  g  Is  from  r”  into  r”*.  Associated  with  this 
problem  Is  the  Wolfe  dual  [22,15] 

maximize  L(x,u)  subject  to  7^L(x,u)  »  0,  u  >  0 
(x.u)€Rn+'" 

(2) 

where  L(x,u):=  f(x)  +  u^g(x) 

Our  penalty  function  Is  derived  from  (2)  by  constructing  an  exterior 
penalty  function  for  the  equality  constraints  only.  Thus  we  define  the 


penalty  function 


0(x,u,y):=  L(x,u)  -  Jl|Vj^L(x,u)Il‘ 


and  consider  the  penalty  problem 


maximize  0(x,u,y) 
(x,u)eR”‘”'’ 
u>0 


which  Is  differentiable  on  R"  when  f  and  g  are  differentiable  on 
r'™.  We  shall  relate  various  stationary  and  solution  points  of  problems 
(1),  (2)  and  (4).  We  begin  with  a  simple  but  useful  result. 


1 
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Theorem  1  (Equivalence  of  stationary  points  of  (1),  (2)  and  (4)) 

Let  f  and  g  be  twice  continuously  differentiable  at  x.  Then 


(a)  Ax,u)  is  a  stationary 
/  point  of  (2)  and 
exists 


(x,u)  is  a  KK' 
point  of  (1) 


/x.u)  is  a  stationary 
point  of  (4)  for 


any  y 


% 


•1 


(x,u}  is  a  stationary 
point  of  (2)  j 


(x,u)  is  a  Kicf 
point  of  (1 )  j 


{x,u)  is  a  stationary  ' 
point  of  (4),  Y  0 
and  ^  is  not  an 
eigenvalue  of  7j^jjL(x,u)/ 


Proof 

The  proof  follows  directly  by  writing  the  Karush-Kuhn-Tucker  conditions 
[15]  (V),  (2')  and  (4')  for  problems  (1),  (2)  and  (4)  respectively  as  follows 

V  L(x,u)  =  0,  g(x)  <  0,  u^g(x)  =  0,  u  >  0  (V) 


For  some  vcR  : 

-  Vj^j^L(;.u)v  *  0 

g(x)  -  Vg(x)v  <  0 
u^(g(x)  -  Vg(x)v)  *  0 
u  >  0 

Vj^L{x,u)  »  0^ 


I 


1  ' 

■] 

I  ( 
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(I  -  yVj^j^L(x,u))7j^L(x.u)  »  o\ 

g(x)  -  YVg{x)7j^L(x,ii)  i  0  \ 
u^(g(x)  -  Y7g(x)7j^L(x,u))  *  0  / 

u  >  0/ 


□ 


(4') 


In  the  next  result  we  establish,  under  appropriate  assumptions,  the 
local  concavity  of  9(x,u,y)  in  both  the  variables  x  and  u. 

Theorem  2  (Negative  semi  definiteness  and  definiteness  of  V  9(x,u,y)) 

Let  (x,u)  be  a  KKT  point  of  (1),  let  f  and  g  be  twice  contin¬ 
uously  differentiable  at  x  and  let  7j^j^L(x,u)  be  positive  definite 

with  minimum  eigenvalue  p  >  0.  Then  for  y  (x,u)  is  a  stationary 

2  _  ^ 

point  of  (4)  and  the  Hessian  7  9(x,u,y)  with  respect  to  (x,u)  is  negative 

semidefinite.  If  in  addition  y  ^  and  7g(x)  has  linearly  independent 
2  -  - 

rows,  then  7  9(x,u,y)  is  negative  definite  and  hence  {x,u)  is  a  strict 
local  maximum  of  (4). 


Proof 

By  Theorem  1,  (x,u)  satisfies  the  KKT  conditions  (4')  for 

problem  (4).  We  have  from  (3)  when  f  and  g  are  differentiable  at  x 
that 


79(x,u,y) 


(I  -y7^L(x,u))7xL(x,u} 
g(x)  -  Y7g(x)7^L(x,u) 


(5) 


Recalling  that  7j^L(x,u)  =  0  we  have  that 


I 
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7^9(x,u,y)  “ 


V  L(x,u)(r  -  (I- Y^vyL(x.u))7g(x) 


Vg(x)(I-YV^j^L(x.u)) 


-Y7g(x)7g(x)^ 


Define 


C:=  7^j^L(x,u)  and  A:*  7g(x) 


C(I-yC)  (I-yC)A^  pc  A^l  Pel  [c 


7^0(x,u,y)  = 


A(I-yC) 


-yAA'J  U  0 


and  for  y  >  we  have  that 

"  p 


(x^  u^)7^e(x,u,Y)(y)  =  x'^’cx  +  ex'^Au  -  yI|Cx+a''’u||^ 

»  -x^Cx  +  2x^{Cx+a'’^u)  -  y|1Cx+a''^u1|^ 

<-p11x1|2+ 211x11  HCx+A^uH-yIICx+A^uH^ 

'  -p((|x||-'^(ICx+A^u|I)^-  (y-|)|1Cx+A^u||^ 

<  0 

Hence  7  0(x,u,y)  Is  positive  semidefinite  for  If  (J)  ^  0  then 

we  consider  two  cases: 

Case  I:  Cx  +  A^u  0.  For  this  case  It  follows  from  Y  >  i  that 

(x^  u^)7^9(5i,G,Y)(JJ)  <  0. 
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Case  II:  Cx  +  A  u  =  0  and  (^)  j*  0.  For  this  case  we  have  that  x  f*  0, 
else  u^A  =  0,  u  0,  which  contradicts  the  assumption  that  the  rows  of 
A  are  linearly  independent.  Hence 

(x^  u^)V^0(x,u.Y)(y)  =  -x'^^Cx  <  0 

where  the  last  inequality  follows  from  the  assumption  that  C  is 
positive  definite. 

T  T  2  X 

Thus  in  either  case  (x  u  )7  0(x,u,y)(|^)  <  0  for  (x,u)  j*  0  and 

V  0(x,u,y)  is  negative  definite  for  Y  >  ^  and  (x,u)  is  a  strict  local 

maximum  of  (4)  [6,13].  □ 

The  assumption  in  Theorem  2  that  Vg(x)  has  full  row  rank  is 
restrictive,  but  apparently  it  is  the  best  we  can  do  if  we  require  that 

V  0(x,u,y)  be  negative  definite.  A  natural  relaxation  is  to  merely  ask 
for  conditions  that  ensure  that  (x,u)  is  a  strict  local  maximum  of  (4). 
It  turns  out  that  such  a  relaxation  can  be  reflected  in  replacing  the 
linear  independence  of  the  rows  of  Vg(x)  by  the  less  stringent  require¬ 
ment  of  the  linear  independence  of  the  gradients  of  the  active  constraints 
only  as  follows. 

Theorem  3  (Strict  local  maximum  of  0(x,u,y)) 

The  last  sentence  of  Theorem  2  can  be  replaced  by  the  following:  If 
in  addition  Y  >  i  aiid  are  linearly  independent  for  icJ  where 

J  ■  {1|g^{x)«0,  i=l . m} 

then  (x.u)  is  a  strict  local  maximum  of  (4). 


(8) 
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Proof 

Let  Aj  =  7gj(x).  From  the  proof  of  Theorem  2,  by  replacing  A  by 

2  —  —  1 
Aj,  we  have  that  Vjj0(x,u,y)  is  negative  definite  for  Y  >  ^  where 


(9) 


We  establish  now  that  (x,u)  Is  a  strict  local  maximum  by  (4)  by 
establishing  the  second  order  sufficient  optimality  condition  [6,13]. 

Note  from  (5)  that  Vy0(x,u,Y)  *  g(x),  and  since  the  optimal  multiplier 
associated  with  the  nonnegativity  constraint  u  >  0  Is  -Vy0(x,u,Y). 
hence  the  second  order  sufficient  optimality  condition  for  (4)  Is  then 


E  ■  {1|u^»0,  g^(x)<0} 

G  ■  {1|u^*0,  g^(x)=0} 

H  «  {11u^>0,  g^(x)=0} 

Since  J  «  GuH  It  follows  that  the  second  order  condition  (10)  can  be 
rewritten  as 

’  X 

0  f  Uq>0 


■i..  -  !-  ~  * 


4 


t 


Condition  (11)  Is  automatically  satisfied  for  Y  >  ^  because  we  have 
already  established  that  7jj0(x,u,y)  Is  negative  definite  for 
Y  >  □ 

So  far  no  convexity  assumptions  have  been  made  anywhere  and  con¬ 
sequently  all  our  results  are  local  results.  We  can  globalize  some  of 
our  results  If  we  assume  that  f  Is  uniformly  strictly  convex  and  g 
Is  convex  on  In  fact  we  can  show  then  that  for  each  local  solution 

(x(y).  u(y))  of  (4),  x{y)  is  the  unique  global  solution  of  (1).  In 
particular  we  have  the  following. 

Theorem  4  (Stationary  points  of  (4)  as  global  solutions  of  (1)  and  (2)) 
Let  f  and  g  be  convex  and  twice  continuously  differentiable  on 
r",  let 

y^V^f(x)y  >  v|iy||^  for  all  x.yeR”  and  some  v  >  0,  (U 

and  let  Y  >  every  stationary  point  (x(y).  u(y))  of  (4),  x(y) 

Is  Independent  of  y  3nd  x(y)  *  x,  where  x  Is  the  unique  solution 
of  (1). 


Proof 

For  X,  y  In  r”  and  ucR*",  u  ^  0  we  have  that 

y^VxxL(x,u)y  >  yv^f(x)y  >  v||y||^  (13) 

Hence  Vj^j^L(x,u)  Is  positive  definite  for  all  u  ^  0  and  Its  smallest 
eigenvalue  p(x,u)  satisfies  the  Inequality  p(x,u)  >  v.  By  Theorem  1(b) 
every  stationary  point  (x(y).  u(y))  satisfies  the  KKT  conditions  (!')  of 
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(1).  Since  f  is  strictly  convex  and  g  is  convex,  x(y)  must  equal 
the  unique  solution  x  of  (1)  and  (x,u{y))  must  solve  (2)  [15].  0 

We  note  that  problem  (4)  can  be  used  directly  to  construct  an 
algorithm  for  solving  the  original  problem.  For  example  we  can  easily 
prescribe  a  Levitin-Poljak  gradient  projection  algorithm  [14]  or  a 
superlinearly  convergent  quasi-Kewton  algorithm  [10,7,9,11,21].  The  key 
observation  to  make  here  is  that  the  projection  operation  here  is  an 
extremely  simple  one,  namely  projection  on  r”xR™.  We  give  below  the 
simplest  gradient  projection  algorithm  for  solving  (4)  and  its  convergence 
to  a  KKT  point  of  (1). 

Algorithm  1  (Gradient  projection  algorithm  for  (4)) 

Choose  Y>0  and  any  {x®,u®) «  R"  xr'J.  Having  (x^,u^)  compute 
as  follows: 

Direction  choice:  p^  *  (I-yVjjjjL(x^,u^))V^L(x^,u^) 

q^  -  (uS(x^)-YVg{x^)V^L(x^u^))+-u^ 

Stepsize  choice:  (x^'*’\u^'*^^)  «  (x^+X^p^,u^+X^q^) 

where  Is  chosen  such  that 

0(x^+X^p^ ,u^+X^q^ ,y) *  max{0(x^+Xp^ ,u^+Xq^ ,y) 

X 

where  6  is  defined  by  (3), 

By  standard  convergence  results  [14]  and  by  Theorem  1  we  have. 

Theorem  5  (Convergence  of  Algorithm  1) 

Let  f  and  g  be  thrice  differentiable  on  r”.  Each  accumulation  point 
(x,u)  of  the  sequence  {(x^.u^)}  generated  by  the  gradient  projection  Algo¬ 
rithm  1,  such  that  ~  is  not  an  eigenvalue  of  Vj^j^L(x,u),  is  a  KKT  point  of  (1). 
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3.  The  Quadratic  Progranroing  Problem 

In  this  section  we  specialize  our  results  to  the  quadratic  programming 
problem  and  obtain  some  sharper  results.  Hov/ever  the  principal  purpose  of 
this  section  is  to  describe  an  SOR  method  for  solving  the  quadratic  program¬ 
ming  problem  which  does  not  require  the  inversion  of  the  matrix  defining  the 
quadratic  term  [17].  This  should  substantially  widen  the  applicability  of 
SOR  methods  to  mathematical  programming  problems  which  have  hitherto  been 
limited  principally  to  the  minimization  of  quadratic  functions  on  the  nonnega 
tive  orthant  [16,17,18].  The  principal  advantages  of  SOR  methods  are  their 
ability  to  handle  extremely  large  problems  and  to  preserve  sparsity. 

We  shall  consider  here  the  quadratic  program 

minimize  yx^Cx  +  d^x  subject  to  Ax  <  b  (14) 

X€R" 

where  C  is  an  nxn  symmetric  matrix,  A  is  an  m^n  matrix,  d  is  in 
r”  and  b  is  in  r"*.  The  dual  to  this  problem  obtained  from  (2)  is 

maximize  ix^Cx  +  d^x  +  u^(Ax-b)  subject  to  Cx  +  d  +  A^u=0,  u>0  (15) 

(x,u)cR"+'"  ^ 

We  note  in  passing  that  the  standard  quadratic  programming  dual  [5,15] 
obtained  by  substituting  from  the  equality  constraint  into  the  objective 
function  of  (15) 

maximize  -ix^Cx-b^u  subject  to  Cx+d  +  A^u*  0,  u>0  (16) 

(x.u)«Rn+m  ^ 

cannot  be  used  to  obtain  a  differentiable  exact  penalty  function  because 
the  optimal  multiplier  associated  with  the  equality  constraint  in  (15) 
is  zero  when  C  is  nonsingular,  whereas  it  is  equal  to  x  in  (16)  also 
when  C  is  nonsingular  [15]. 
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The  penalty  function  associated  with  (15)  Is 

<|>(x,u.y):*  |■x^Cx  +  d^x  +  u^(Ax-b)  -  Jl|Cx+A^u+d||^  (17) 

and  the  associated  penalty  problem  Is 

maximize  (i>(x,u>Y)  (18) 

(x,u)eR"‘™ 
u  >  0 

We  have  as  an  Immediate  consequence  of  Theorems  2  and  3  the  following. 

Theorem  6  (Concavity  and  strict  concavity  of  (J)(x,u,y)) 

Let  C  be  positive  definite  with  minimum  eigenvalue  p  >  0.  Then 
1  2 

for  ^  <l>(XtU,Y)  Is  negative  semidefinite  and  hence  (j>(x,u,Y) 

Is  a  concave  function  of  (x,u)  on  If  In  addition  y  >  i  and 

2  ^ 

A  has  linearly  Independent  rows,  then  V  (J>(x,u,y)  Is  negative  definite 
and  hence  <ti(x,u,Y)  Is  a  strictly  concave  function  on  r”"*™.  If 
Y  >  ^  and  only  A^,  IcJ  are  linearly  Independent  where 

J  =  {1lA^x*b^,  1=1,..., m} 

and  (x,u)  Is  a  KKT  point  of  (1),  then  (x,u)  Is  a  strict  global 
maximum  solution  of  (18). 

Corollary  1  Let  {x|Ax^b}  be  nonempty,  let  C  be  positive  definite 
with  least  eigenvalue  p  >  0.  Then  for  each  problem  (18)  Is 

a  concave  quadratic  maximization  problem  which  possesses  a  solution 
(x(y)»  u(y))  with  x(y)  Independent  of  y  and  x(y)  ■  x  where  x  Is 
the  unique  global  solution  of  (14). 


With  the  help  of  the  SOR  scheme  of  [16]  we  can  solve  iteratively  the 
quadratic  program  (18)  in  r”xR™  and  thereby  obtain  a  solution  to  (14). 
It  will  be  convenient  for  that  purpose  to  have  the  following  expressions 


at  hand 


V4»(x,u,y) 


V^<>(x.u,y)  = 


(I-YC)(Cx+A'u+d) 


Ax  “  b  -  yA(Cx+A  u+d 


C(I-yC)  (I-yC)A^ 


A(I-yC) 


An  SOR  method  for  solving  the  quadratic  program  (18)  with  relaxation 
factor  u)€(0,2)  can  be  given  as  follows  then 


* - n — 1 - . ’^j-l’^j’  *  *  ‘  ’ 


j*l f • • • • f n 


u]*’.(u]h - V-I - v„ 


. 


J*1 f  * .  .  • 


We  spell  out  our  SOR  scheme  in  detail  now. 


Algorithm  2  (SOR  scheme  for  (18)) 


Choose  u)€  I 


(0.2),  T  >  MX - li-  C,  ^O  ’, 


an  eigenvalue  of  C 


and  (x*^,u®)€  R^xR".  Having  (x^u^)  compute  (x^‘*’\u^‘*‘b  as  follows; 


•■ir 


i 


- ; - r2(^^1-YCi)(  I  C.oxi  '+  I  C..x;+A'u'+d)) 

Cjj-YiiCjir  J  J  i-1 


ror  j  > 


j*lf«***9n 


“]*’  *  <“1*“^* - 2(Ajx'*'-b,-YA,(Cx’*'tV(A’’)..ul*’+  I  (A^).  tlij+d)))+ 

■>  J  -TllAjlr  ■’  ■'  J  1=1  .  ‘  *  t-j  ‘  * 


For  j  >  1  only 

j*l f • • • • jW 


Remark  1 


The  only  implicit  assumption  in  Algorithm  2  is  that  Aj  j*  0, 

J»1 . .  This  assumption  imposes  no  restrictions  whatsoever,  since 

all  constraints  A^x  ^  of  (14)  for  which  Aj  *  0  are  either 
inconsistent  (bj<0)  or  else  can  be  discarded. 


Remark  2 

Note  that  in  Algorithm  2  only  linear  arrays  are  needed  in  distinc¬ 
tion  from  rectangular  arrays.  That  is,  we  need  to  access  the  rows  and 
columns  of  C  and  A  one  at  a  time.  Thus,  if  the  problem  is  of 
enormous  size  and  very  sparse,  then  only  the  nonzero  elements  need  be 
stored,  and  this  sparsity  unlike  pivotal  algorithms  is  never  lost. 

We  can  now  use  the  convergence  theorems  of  [16]  and  the  theorems 
of  this  paper  to  obtain  the  following  convergence  result  for  the  SOR 
Algorithm  2. 

Theorem  7  (Monotonicity  and  convergence  of  the  SOR  Algorithm  2) 

For  the  sequence  {(x^,u^)},  i*l,2,..,,  generated  by  Algorithm  2 

♦{x^'*‘\u^'*‘\y)  >  ♦(x^u^Y)t  1“0,1, -  (J 
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and  each  accumulation  point  (x,u)  of  the  sequence  {(x^.u^)}  is  a  KKT 
point  of  the  original  quadratic  program  (14).  If  in  addition  C  is 
positive  semi definite  then  x  is  a  global  solution  of  (14). 

Proof 

Inequality  (22)  follows  from  (9)  of  [16]  and  by  Theorem  2.1  of  [16] 
(x,u)  is  a  stationary  point  of  (17).  By  Theorem  1,  (x,u)  is  a  KKT 
point  of  (14).  When  C  is  positive  semidefinite  x  is  a  global  minimum 
solution  of  (14)  by  the  sufficiency  of  the  KKT  conditions  [15].  □ 

We  note  that  Theorem  7  does  not  ensure  the  existence  of  an 
accumulation  (x,u)  of  the  sequence  {(x^,u^)}  of  Algorithm  2.  To 
ensure  that  at  least  one  accumulation  point  exists  we  need  to  impose 
some  sort  of  qualification  similar  to  that  of  Theorem  2.2  of  [16]  which 
will  ensure  the  boundedness  of  the  iterates  {(x  ,u  )}  of  Algorithm  2. 

In  particular  we  have  the  following. 

Theorem  8  (Boundedness  of  the  iterates  of  the  SOR  Algorithm  2) 

Let  C  be  positive  definite  with  minimum  eigenvalue  P  >  0,  let 
A  have  linearly  independent  columns,  let  x  satisfy  the  constraint 

qualification  Ax  <  b  and  let  y  >  4-.  Then  the  sequence 

i  i  ^ 

{(x  ,u  )},  i=l,2,....,  generated  by  the  SOR  Algorithm  2  is  bounded 

and  lim  x^  *  x,  where  x  is  the  unique  global  solution  of  (14). 

i->oo 

Proof 

2 

By  Theorem  6  the  constant  Hessian  V  (j»(x,u,Y)  defined  by  (20)  is 
negative  semidefinite.  We  shall  assume  that  the -sequence  {(x^,u^)} 


generated  by  Algorithm  2  Is  unbounded  and  exhibit  a  contradiction.  With¬ 
out  loss  of  generality  suppose  that  ll(x^,u^)l|  f  0  and  {||x^,u^  . 


Define  z;**  (y),  M:»  V^<|)(x,u,y)  and  q:= 


y 

‘’i 

:■ 

d-YC)d 

*^2 
V  J 

-b-YAd 

1  4 

.  Then 


♦(x.u.y)  »  <I>(z.y)  *  +  q^z 

It  follows  from  (22)  and  Algorltfmi  2  for  1=1,2 . .  that  u^  ^  0  and 


pY  “  iiir 


1  z 
? 


1 

ii?ii 


.T  z^ 


By  the  Bolzano-Welerstrass  Theorem  we  get  that 


f  \ 


has  an 


accumulation  point  y  on  the  unit  sphere  In  satisfying  0  <  l-y^My 
and  y  =  (5)  with  xcR”  and  ueR^.  Since  M  Is  negative  semidefinite 
It  follows  that  y^My  *  0  and  hence  My  =  0.  Since  we  also  have  that 


.  1  ilV  ^  gV  <  gV 
Hz’ II  "  lU'll  ^ IzT  M‘llz’ll 

It  follows  that  0  ^  q^y.  We  thus  have 

-  0.  q^y  >  0.  0  j*  y  -  (§).  u  >  0  (23) 


or  equivalently 
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T1 


C(I-yC)  (I-yC)A 
[a(I-yC)  -yAA^J 


(q)  =  0.  qjx  +  qju  >  0,  u  >  0,  (x,u)  i  0 


(24) 


From  the  generalized  Gordan  theorem  of  the  alternative  [19]  (24)  is 
equivalent  to  either 

the  rows  of  [C(I-yC)  (I-yC)A^3  are  linearly  dependent  (25) 

or 

C(I-yC)v  +  (I-yC)A^w  =  (I-YC)d  \ 

A(I-yC)v  -  yAA^w  >  -b-YAd  \  (26) 

has  no  solution  (v,w)  in 

Because  Y  >  ^  it  follows  that  I  -  yC  is  negative  definite  and  that 
C(I-yC)  is  nonsingular  which  contradicts  (25).  We  will  show  now  that 
(26)  also  leads  to  a  contradiction.  By  hypothesis  we  have  that  Ax  <  b. 
Since  the  columns  of  A  are  linearly  independent,  there  exists  a  w 
satisfying 

A^w  =  d  +  Cx 


and  hence 

Ax  »  AC'^AVd)  <  b 


that  is 


AC*^d  -  AC"^a\  +  b  >  0 


or 

AC"^(I-YC)d  -  (I-yC)a'^w)  -  yAA^w  >  -  b  -  YAd 


By  defining 


V  -  (I-YC)"’c"^((I-YC)d- (I-yC)a'^w) 

we  get 

C(I-yC)v+(I-yC)a'’*w=  (I-YC)d 
A(I-yC)v  -  yAA^w  >  -  b  -  YAd 

These  last  two  relations  contradict  (26).  Consequently  the  sequence 
{(x\u^)}  Is  bounded  and  must  have  at  least  one  accumulation  point.  For 
each  accumulation  point  (X.O),  X  must  equal  the  unique  solution  x  of 
(14).  Since  {x^}  Is  also  bounded  It  must  converge  to  x  [2].  □ 

At  this  time  we  do  not  have  any  computational  experience  for  the  SOR 
Algorithm  2  for  solving  the  general  quadratic  programming  problem  (14). 
However,  for  the  case  when  matrix  C  ®  el  where  e  is  a  positive  number 
and  Y  ■  the  penalty  problem  (18)  becomes 

Maximize  -  i  II A^u+d  11^  -  eb^u  (27) 

ueR*"  ^ 
u>0 

This  Is  precisely  the  dual  of  the  quadratic  program  perturbation  of  [17] 
associated  with  the  linear  program 

Minimize  d^x  subject  to  Ax  <  b  (28) 

xeR” 

and  which  was  solved  quite  successfully  by  the  SOR  method  proposed  here. 
Thus  for  at  least  this  special  class  of  quadratic  programs  computational 
experience  Is  very  encouraging.  It  Is  hoped  that  this  experience  will 
carry  over  to  the  more  general  case. 
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